METHOD AND APPARATUS FOR SINGLE IMAGE 3D 
VISION GUIDED ROBOTICS 

Cross-Reference to Related Applications 

5 

This application is a continuation-in-part of United States patent 
application no. 10/153680, filed May 24, 2002 which is pending^ 

Technical Field 

lO 

The invention relates to the field of vision guided robotics, and 
more particularly to a method and apparatus for single image three dimensional 
vision guided robotics. 

15 Background 

Robots have long been widely used in manufacturing processes 
for many applications. Many different types of sensors are used to guide robots 
but machine vision is increasingly being used to guide robots in their tasks. 

20 Typically such machine vision is used in a two-dimensional application wherein 
the target object need only be located in an x-y plane, using a single camera. 
For example see United States patent no. 4,437,1 14 LaRussa. However many 
robotic applications require the robot to locate and manipulate the target in 
three dimensions. In the past this has typically involved using two or more 

25 cameras. For example see United States patent no. 4,146,924 Birk et al.; and 
United States patent no. 5,959,425 Bieman et al. In order to reduce hardware 
costs and space requirements it is preferable to use a single camera. Prior 
single camera systems however have used laser triangulation which involves 
expensive specialized sensors, must be rigidly packaged to maintain geometric 

30 relationships, require sophisticated inter-tool calibration methods and tend to be 
susceptible to damage or misalignment when operating in industrial environ- 
ments. 



35 



Target points on the object have also been used to assist in 
determining the location in space of the target object using single or multiple 
cameras. See United States patents no. 4,219,847 Pinkney et al. and United 
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States patents no. 5,696,673; 5,956,417; 6,044,183 and 6,301,763 all of Pryor 
and United States patent no. 4,942,539 of McGee et al. Typically these 
methods involve computing the position of the object relative to a previous 
position, which requires knowledge of the 3D pose of the object at the starting 
5 point. These methods also tend to not provide the accuracy and repeatability 
required by industrial applications. There is therefore a need for a method for 
calculating the 3D pose of objects using only standard video camera equipment 
that is capable of providing the level of accuracy and repeatability required for 
vision guidance of robots as well as other applications requiring 3D pose 
1 0 information of objects. 

Summary of Invention 

1 5 A method of three-dimensional object location and guidance to 

allow robotic manipulation of an object with variable position and orientation 
by a robot using a sensor array is provided, the method comprises: a) calibrat- 
ing the sensor array to provide a Robot - Eye Calibration by finding the intrinsic 
parameters of said sensor array and the position of the sensor array relative to a 

20 preferred robot coordinate system ("Robot Frame") by placing a calibration 
model in the field of view of said sensor array; (b) training object features by: 
i) positioning the object and the sensor array such that the object is located in 
the field of view of the sensor array and acquiring and forming an image of the 
object; ii) selecting at least 5 visible object features from the image; iii) 

25 creating a 3D model of the object ("Object Model") by calculating the 3D 
position of each feature relative to a coordinate system rigid to the object 
("Object Space"); (c) training a robot operation path by: (i) computing the 
"Object Space -> Sensor Array Space" transformation using the "Object 
Model" and the positions of the features in the image; (ii) computing the 

30 "Object Space" position and orientation in "Robot Frame" using the transfor- 
mation from "Object Space-> Sensor Array Space" and "Robot - Eye Calibra- 
tion"; (iii) coordinating the desired robot operation path with the "Object 
Space"; (d) carrying out object location and robot guidance by: 
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(i) acquiring and forming an image of the object using the sensor array, search- 
ing for and finding said at least 5 trained features; 

ii) with the positions of features in the image and the corresponding "Object 
Model" as determined in the training step, computing the object location as the 
5 transformation between the "Object Space" and the "Sensor Array" and the 
transformation between the "Object Space" and "Robot Frame"; (iii) 
communicating said computed object location to the robot and modifying robot 
path points according to said computed object location. 

1 0 The invention further provides a system for carrying out the 

foregoing method. 

Brief Description of Drawings 

15 In drawings which illustrate a preferred embodiment of the 

invention: 

Fig. 1 is a perspective view of a vision-guided robot; 

Fig. 2 is a schematic diagram illustrating the relative frames of 
reference for calculating the position of the object; 
20 Fig. 3 is a schematic diagram illustrating the calculation of the 

intended operation path on the object; 

Fig. 4 is a representation of the operator's computer screen for 
selecting reference features on the object; 

Fig. 5 is a flow chart illustrating the calibration of the camera 
25 mounted on the robot arm; 

Fig. 6 is a flow chart illustrating the method of teaching the object 
features and handling path; and 

Fig. 7 is a flow chart illustrating the method of object positioning 

and handling. 

30 

Description 



Throughout the following description, specific details are set 
forth in order to provide a more thorough understanding of the invention. 



- 4 - 



However, the invention may be practiced without these particulars. In other 
instances, well known elements have not been shown or described in detail to 
avoid unnecessarily obscuring the invention. Accordingly, the specification 
and drawings are to be regarded in an illustrative, rather than a restrictive, 
5 sense. 

Fig. 1 shows a robot 10 having a manipulating arm 12, base 22 
and attached tool 14 with camera 16 mounted thereon. Tool 14 is designed to 
manipulate a part or target object 18. Camera 16 is preferably a commercially 
1 0 available analog or digital video camera. Camera 16 and robot 10 are electri- 
cally connected to computer control station 24 for communication of data back 
and forth. 

The method is performed in the main steps described as follows: 

a) calibration of the camera; 

b) teaching the features on the object; and 

c) finding the three-dimensional pose of the object and using 
this information to guide the robot to approach the object 
to perform any operations (e.g. handling, cutting etc.). 

In the following discussion the following terms have the foll- 
owing meanings, as illustrated in Fig. 2: 

25 Transformation: three-dimensional rotation & translation be- 

tween two spaces; 

Tool 14: the tool the robot is using for performing the handling, 
cutting or other robotic operations, having an operating end or 
"end-effector"; 

30 Camera Space 26: a reference frame defined with respect to a 

point on, and therefore rigid to, the camera 16; 
Tool Space 28: a reference frame defined with respect to a point 
on, and oriented along the direction of the end-effector and there- 
fore rigid to, the tool 14; 
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20 



) 
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Robot base Space 30: a reference frame defined with respect to 
the physical base of the robot arm and therefore rigid to, the robot 
base 30; 

Robot Frame: a preferred robot coordinate system 
5 Calibration Model or Calibration Pattern or Calibration Template: 

a physical object with features of whose relative geometry is 
known 

Sensor Array: a collection of one or more sensors capable of 
forming a single image 
1 0 Training Space 32: a reference frame defined with respect to a 

point on the calibration template 40, and aligned to its main axes; 
Object Space 36: a reference frame defined with respect to, and 
therefore rigid to, the object 18; 

Object Frame 34: a reference frame defined with respect to a 
1 5 point on, and therefore rigid to, the object 18; 

Camera calibration intrinsic parameters: focal length, image 
center, real pixel size, and radial and tangential distortion for the 
lens; 

Camera calibration extrinsic parameters: rotation and translation 
20 of an external space relative to the Camera Space 26. 

Calibration 

The calibration process involves: i) finding the camera intrinsic 
parameters and ii) the position of the camera relative to the tool of the robot 
("hand- eye" calibration). The position of the camera in the 'Training Space", 
which is a space rigid to the place where the object will be trained is also 
determined. A general explanation of the basic calibration algorithms and 
descriptions of the variables can be found in the following publications: 

"An Efficient and Accurate Camera Calibration Technique 
for 3D Machine Vision", Roger Y. Tsai, Proceedings of 
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IEEE Conference on Computer Vision and Pattern Recog- 
nition, Miami Beach, FL, 1986, pages 364-374; 

"A Versatile Camera Calibration Technique for High- 
5 Accuracy 3D Machine Vision Metrology Using Off-the- 

Shelf TV Cameras and Lenses", Roger Y. Tsai, IEEE 
Journal of Robotics and Automation, Vol. RA-3, No. 4, 
August 1987, pages 323-344. 

1 0 "Tsai's Camera Calibration Method Revisited" 

http://www.ai.mit.edu/people/bkph/papers/tsaiexplain.pdf 

Tsai's camera model is based on the pin-hole model of perspec- 
tive projection. Given the position of a point in 3D world coordinates, the 
1 5 model predicts the position of the point's image in 2D pixel coordinates. Tsai's 
model has 1 1 parameters: five internal (also called intrinsic or interior) parame- 
ters: 

i) f - effective focal length of the pin hole camera; 

20 ii) kappal - 1st order radial lens distortion coefficient; 

iii and iv) Cx, Cy - coordinates of center of radial lens distortion 
and the piercing point of the camera coordinate frame's 
Z axis with the camera's sensor plane; 
v) sx - scale factor to account for any uncertainty due to 

25 framegrabber horizontal scanline resampling, 

and six external (also called extrinsic or exterior) parameters: 

i) Rx, Ry, Rz - rotation angles of the transformation be- 
tween the world and camera coordinate frames, and 

ii) Tx, Ty, Tz - translation components of the transfor- 
30 mation between the world and camera coordinate fra- 
mes. 

The internal parameters describe how the camera forms an image 
while the external parameters describe the camera's pose (i.e. position and 



- 7 - 



orientation) in the world coordinate frame. Calibration data for the model 
consists of 3D (x,y,z) world coordinates of a feature point (in mm. for example) 
and corresponding 2D coordinates (Xf ,Yf) (typically in pixels) of the feature 
point in the image. Two forms of calibration are possible: 
5 coplanar - the calibration points lie in a single plane in 3D, and 

non-coplanar - the calibration points occupy a 3D volume. 

As illustrated in Fig. 2 and 5, the first step in calibration is to 
position the camera 16 on the robot arm 12 so it is orthogonal to a template 40 

lO using the mirror approach, and defining the 'Training Space" for the robot 

aligned with the template 40 used for calibration. A calibration template 40 is 
placed on the floor or other planar surface. It is a flat piece of paper, plastic or 
other medium having a series of fixed detectable features such as a grid of dots 
or squares. The camera 16 is positioned so that it is orthogonal to the plane of 

1 5 the calibration template 40, that is, so the camera's imaging plane is parallel to 
the template 40. The mirror approach for obtaining orthogonality comprises 
placing a flat mirror on the grid of the template 40 and centering the lens on the 
mirror in the center of the grid. 

20 Next the camera intrinsic parameters and the "Camera -> Training 

Space " transformation are computed considering the Training Space. Next, 
the "Camera -> Tool" transformation is computed using the "Camera -> 
Training Space" transformation and inquiring the robot about the "Tool" 
position in "Training Space". To calculate the "Camera -> Tool" transforma- 

25 tion manually, the operator first touches 3 identifiable points on the grid having 
known coordinates with the tool. Next the operator stores images of the grid 
from at least 2 and preferably 4 measured heights above the grid. 

Alternatively, the calibration can be done automatically to 
30 compute both the camera intrinsic parameter and the hand-eye calibration. The 
technique requires the camera to observe a planar pattern shown at a plurality 
of (at least two) different orientations. The pattern can be printed on a laser 
printer and attached to a planar surface. The position of the tool at each station 
is acquired directly from the robot. The operator positions the calibration 
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pattern in front of the robot and camera and starts the procedure. The automatic 
calibration takes place in less than 5 minutes. The calibration can be carried 
out in a different location from the part training or manipulation. The calibra- 
tion pattern can be mounted in a fixed position, out of the working space, and 
5 automatic calibration can take place at regular time intervals. 

The following steps are carried out to perform the automatic 

calibration: 

1 0 a) the calibration pattern is positioned in the field of view of 

the robot mounted camera 16; 

b) the robot is moved to a plurality of stations in a predefined 
manner so that the calibration pattern is in the field of view at each station (see 

15 Roger Y. Tsai and Reimar K. Lenz, "A New Technique for Fully Autonomous 
and Efficient 3D Robotics Hand/Eye Calibration", IEEE Transactions on 
Robotics and Automation, Vol. 5, No. 3, June 1989 p. 345 at p. 350); 

c) At each station the following operations are performed: 
20 - the acquired image is processed to extract the calibration 

points from the image; 

the robot is asked for the tool position; 

d) Using the calibration points information at each station, 
25 calibrate the camera intrinsic parameters and compute the extrinsic transforma- 
tion from the pattern to the camera; 

e) Using the extrinsic transformation at each station and the 
corresponding tool position, the camera to tool transformation is calculated (see 

30 Tsai and Lenz reference above at p. 350). 



Teaching 

Teaching the object is the process of: 



- 9 - 



a) Selection from the object's image of a set of at least 5 
features, and determining the position of the features in the image. Features can 
be edges, holes, corners, blobs (extracted from the image) or simply a region of 
the image which will be used in a pattern match. Preferably, one or more 

5 unique features may be selected to be considered the "anchor features". The 
other selected features may be small, non-unique features relative to the anchor 
features. 

b) Real world coordinates are computed for the selected 

1 0 features. The object is located in the Training Space, so that by using the fea- 
tures 1 heights relative to the Training Space, the 3D position of the object 
features inside the Training Space can be computed using the position in the 
image and the Training Space to Camera Space transformation calculated at 
calibration. 

15 

c) An Object Space is defined such that it is identical to the 
Training Space but is rigid to the object and moves with the object. 

d) Also an Object Frame will be defined with a constant 
relationship to the object, in a position selected by the user. Three non-co- 

20 linear points may be used to define the Object Frame. 

e) The Object Frame (computed in tool coordinates) is sent to 
the robot to be considered as the robot working space. To find this frame 
position, the transformation from Object Space to Camera Space is used, then 

25 from Camera Space to Tool Space. 

f) Relative to the Object Frame, the operator can train the 
intended operation path (the tool path shown in Fig. 3) for the robot by 
moving the robot or alternatively by using an off-line robot programming 

30 software application to create the desired operation path with respect to the 
object. 

As illustrated in Fig 3, 4 and 6, the first step in teaching the 
object features is to place the object in the "Training Space " and capture an 
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image with the robot in the calibration position (where the "Camera Space -> 
Training Space" transformation was calculated). Figure 4 illustrates the image 
50 of the part as displayed on the operator's computer screen. Next, at least 5 
visible features 54 are selected. Preferably at least one anchor feature 52 is 
5 selected from the image and a set of at least 4 other visible features 54 are 

selected and positioned relative to the anchor feature. The 3D position of each 
feature 54 is then calculated in "Training Space" using the height of the feature 
relative to the Training Space and "Camera Space -> Training Space" trans- 
formation. An "Object Space " is then defined as having a constant relationship 

1 0 with the "Training Space" but connected to the object and the 3D coordinates of 
the features are transferred into that space. The "Object Space -> Camera 
Space" transformation is computed using the 3D position of the features inside 
this space and the position in the image by computing an extrinsic calibration 
using the camera calibration from the previous step. Next an "Object Frame" 

1 5 inside "Object Space" is defined to be used for teaching the intended operation 
path. This Object Frame position and orientation in "Tool Frame" is computed 
using the transformation from "Object Frame-> Camera Space" and "Camera 
Space-> Tool Space". The "Object Frame" is then sent to the robot and the 
intended operation path is trained inside this space. 

20 

The following method may also be used for computing the 3D 
position of the selected features 54 automatically, without any prior knowledge 
about the part. The following algorithm is developed using the approach 
described in Guo-Quing Wei,. "Active Self-Calibration of Robotic Eyes and 

25 Hand-Eye Relationship with Model Identification", IEEE Transactions on 

Robotics and Automation, Vol. 14, No. 1, February 1999 p. 158. The camera is 
rigidly mounted on the robot gripper. The derived method computes the world 
coordinates of the features based on robot motion parameters, known camera to 
tool transformation, image coordinate measurements and intrinsic camera 

30 parameters. The robot tool will undergo a set of pure translations from a base 
position P 0 to a set of positions Pj. 



The motion equations for a point p { in the image are: 
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Where: 

5 j represents the index of the station where the measurement takes place; 
R cg are elements of the rotation matrix of the camera to tool transformation; 
t 0 j represents the translation from the current station to the base position; 
u 0 ,v 0 is the camera central point; 

Ujj.Vjj are the undistorted image coordinates of the i-th feature in the image 
lO acquired at station j. To compute these values one uses the already calibrated 
cameras intrinsic parameters. 

At least two stations are needed for the linear system to have 
unique solutions, but a set of at least 3 stations is used in order to compensate 
1 5 for the noise in the images and other perturbation factors that may occur. 

X 0 ,Y 0 ,Z 0 are computed for all the features in camera space, but 
the values can be transposed in any other space that is related to it, such as 
training space, tool space or even robot base space. The space in which the 
20 coordinates are represented makes no difference as this space is only used to 

compute the current transformation to the camera and to transfer then the object 
frame points to the tool space. 

The robot stations are located on a circle around the base to 
25 assure a uniform distribution. 

The automatic feature position finding steps are as follows: 
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a) the part is positioned in front of the camera 16; 

b) the features 54 that are going to be used for 3D part 
positioning are selected; 

c) the automatic feature position finding procedure is started 

5 by: 

i) moving the robot to a set of stations around the base 
22; 

ii) memorizing the tool position relative to base at each 
station; 

1 0 iii) at each station acquiring an image, finding the visi- 

ble features 54 and for those features calculating the 
undistorted image position; 

d) for each feature solve a linear system of as many equations 
as the number of images the given feature was visible in; 

15 e) the calculated positions are transformed in a space that 

suits the application. 



20 



25 



30 



Alternatively the location of the features can be sourced by using 
a CAD model of the part. 

1. Object Location & Robot Guidance 

To carry out object location and robot guidance, the following 
steps are carried out: 

a) The tool 14 is positioned in any predefined position above 
the bin with objects 18. 

b) An image of the object 18 is captured. 

c) The trained features 54 are searched for. If any anchor 
features were selected, then a first search is done for the anchor features 52. 
Using the position and orientation of the anchor features 52 the rest of the 
features 54 can be found by their relative positions. This approach allows 
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similar features to be selected as features are searched in a relatively small 
region of interest. Otherwise, each feature is searched over the entire image. 

d) The position (in the image 50 and in the Object Space) of 
5 the found features (at least 5) are used to calculate the transformation between 

the Object Space and the Camera Space using an extrinsic calibration algo- 
rithm. (See the Tsai article above). The found position is used to re-orient the 
camera to "look" at the object from an orthogonal position which is the one 
used at training. This last step may be necessary if the object has major rota- 
1 0 tions, since in this case the features may be distorted and the found position 
may not be completely accurate. 

e) Steps c) and d) above are repeated. 

15 f) The previous "Object Space to Camera Space" transforma- 

tion is used in conjunction with the "Camera Space to Tool Space" transforma- 
tion to find the position of the Object Frame in Tool Space. 

g) The Object Frame is then sent to the robot to be used as 
20 the reference frame for the robot's operation path. 

With reference to Fig. 7, the preferred steps to carry out object 
location and robot guidance are illustrated. First the robot is positioned in a 
predefined position above the bin or container that holds the target object. If 

25 no object is in the field of view, the robot is moved until at least one anchor 

feature is found in the image. Next, the position and orientation of the anchor 
feature is used to compute the position of the rest of the features. The position 
of all the visible features in the image is found. With the positions of features 
from the image and their corresponding positions in "Object Space" (calculated 

30 during teaching) the camera calibration is used to compute the transformation 
between the "Object Space" and "Camera Space". The camera extrinsic 
calibration is used as described in the Tsai article referenced above. 
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Next the transformation described above is used to calculate the 
movement of the robot to position the camera so that it "looks" orthogonally at 
the object, namely the same position as in training. In this way all the features 
will appear as similar as possible as at training. This will make the recognition 
5 and positioning more accurate. Next the "Object Space -> Camera Space" 
transformation is found in the same way as in the previous step (using the 
features positions). The Object Frame memorized at training is computed using 
the found transformation and "Camera Space-> Tool Space" transformation. 
Next, the commuted "Object Frame" is sent to the robot. The "Tool" position is 
1 0 used to define the frame in "Robot Space". The trained robot path is performed 
inside this space. 

Thus methods for teaching robots and handling of objects by 
robots in three dimensions using one camera mounted on the robot arm are dis- 

1 5 closed in which targets are used on objects. The targets are usually normal 

features of the object or could be created using markers, lighting, etc. It is not 
necessary to use the CAD model of the object according to this method. The 
objects need not be fixtured and can be placed anywhere within the workspace 
of the robot. While the method has been described for one trained object, the 

20 process can be used in the same manner to first recognize the object and then 
find its location in three dimensions. Also the method has been described for 
one visible set of features selected on one side of the object, but it can be 
extended to all the sides that can appear in a working situation. 

25 In the method described above, the calibration step can be carried 

out with just the intrinsic and hand-eye calibration, and in that case, in teaching 
step b), the 3D position of features may be provided using the CAD model of 
the object. Similarly the 3D position of features can be calculated automati- 
cally, without any model or measurements on the part. 
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Further, training step c) can be accomplished by sending to the 
robot the positions of intended robot path points, with the positions computed 
using the object's current position and orientation. The robot path points can be 
sourced from an offline robot programming software application. Also, one or 
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more features on the object can be used as grasp points and the position of 
those points in robot coordinates sent to the robot to eliminate the need for 
manually teaching grasp points using the robot teach pendant. Alternatively the 
coordinates of the object frame points or other points of interest (e.g. robot path 
5 points) can be transformed using the transformation from tool to robot base and 
all the coordinates send to robot in the robot base coordinate frame instead. 

Further, the calibration and teaching steps a) and b) can be 
combined by using a self calibration method of robotic eye and hand-eye 

1 0 relationship with model identification as described in "Active self-calibration 
method of robotic eye and hand-eye relationship with model identification " by 
Guo-Qing Wei, Klaus Arbter and Gerd Hirzinger. The result of such a method 
will give camera intrinsic parameters, hand-eye calibration and position of 
selected features in camera space; The rest of the path training and run time 

1 5 will still remain the same and in this preferred approach; 

In accordance with the present invention the determination of 
object location in step c(i) and d(ii) can use any of the following algorithms: 

20 a) 3D pose estimation using non linear optimization methods 

derived from the ones described in the already mentioned articles : 

"A Flexible New Technique for Camera Calibration " by 
Zhengyou Zhang; 

"An efficient and Accurate Camera Calibration Techniques 
25 for 3D Machine Vision" by Roger Y. Tsai; 

b) 3D pose estimation from lines correspondence ( in which case 
selected features will be edges ) as described in "Determination of Camera 
Location from 2D to 3D Line and Point Correspondences" by Yucai Liu, Thomas 
30 S. Huang, Olivier D. Faugeras; 



c) pose estimation using "orthogonal iteration" described in 
"Fast and Globally Convergent Pose Estimation from Video Images" by 
Chien_Ping Lu, Gregory D. Hager, Eric Mjolsness; 
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d) approximate object location under weak perspective 
conditions as demonstrated in "Uniqueness of 3D Pose Under Weak Perspective: 
A Geometric Proof by Thomas Huang, Alfred Bruckenstein, Robert Holt, Arun 
Netravali; 

5 

e) approximate object location using Direct Linear Transforma- 
tion ( DLT) as described in "An investigation on the accuracy of three-dimensional 
space reconstruction using Direct Linear Transformation techniques" by Chen, 
Armstrong, Raftopoulos. 

lO 

In addition to use in robotics, the described method can be applied 
to a variety of industrial and non-industrial processes whereby knowledge of the 
3D pose of an object is required. V 

1 5 While the invention has been described using a single camera, the 

image may be formed using multiple sensors ("Sensor Array"). For instance, the 
formation of a single image may be accomplished using multiple cameras that are 
situated such that the origins of their camera spaces are coincident. In this fashion, 
each camera would view a different area of the part with some overlap with the 

20 areas viewed by adjacent cameras. A single image is then formed by "Mosaicing" 
the images from individual cameras similar to one described in Multiple View 
Geometry in Computer Vision by Richard Hartley and Andrew Zisserman, 
Cambridge University Press, 2000. The same image formation may be accom- 
plished by mounting a single camera on a robot and rotating the camera about its 

25 origin of the camera space and capturing the multiple images needed for 
Mosaicing. The object features may be extracted from multiple images captured 
by the same sensor array located in the same position whereby each image is 
formed under a different combination of lighting and filters to highlight a group 
of object features that are not apparent in other images. The object features 

30 themselves may be created using markers, lighting or other means. 

The calibration and teaching steps may be accomplished by using 
a self calibration method of robotic eye and hand-eye relationship with model 
identification. 
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The creation of the Object Model may be accomplished by using the 
relative heights of features and the "Robot - Eye Calibration" or by the operator 
entering the 3D position of each feature manually from a CAD model, measure- 
5 ment or other source. 

While the invention has been described with the camera mounted on 
the robot, the same approach is valid for cases whereby the camera is fixed onto 
a stationary structure. Similarly although the object has been described to be at 
1 0 rest on a surface, the same approach is valid for cases when the object is stationary 
but grasped by a robot. Also during the operation of the system there may exist 
a relative velocity between the sensor array and the object wherein step d) is 
executed in a continuous control loop and provides real time positional feedback 
to the robot for the purpose of correcting the intended robot operation path. 

15 

Where the camera is mounted on the robot, the calibration step is 
accomplished by: i) the operator moving the "Calibration Model" relative to the 
camera and capturing images at multiple positions to determine camera intrinsic 
parameters; ii) the operator taking an image of the "Calibration Model" at a 

20 stationary position to determine the extrinsic calibration parameters; iii) the 
operator determining the position of the "Calibration Model" in the robot space 
using the robot end-effector while the "Calibration Model" is at the same position 
as in step ii); and iv) calculating the "Robot - Eye Calibration" using results of i), 
ii), iii). The step of creating the Object Model is accomplished by using the 

25 relative heights of features and the "Robot - Eye Calibration". 

Where the camera is fixed onto a stationary structure and the object 
is at rest upon a surface, the calibration step is accomplished by mounting the 
"Calibration Model" on the robot and using the robot to automatically move the 
30 "Calibration Model" relative to the camera and capturing images at multiple 
known robot positions. The step of creating the Object Model is accomplished by 
the operator entering the 3D position of each feature manually from a CAD model, 
measurement or other source. 
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Where the camera is fixed onto a stationary structure and the object 
is in a robot's grasp such that the position and orientation of the object can be 
modified by known values, the calibration step is accomplished by mounting the 
"Calibration Model" on the robot and using the robot to automatically move the 
5 "Calibration Model" relative to the camera and capturing images at multiple 
known robot positions. The step of creating the Object Model is accomplished 
by using the robot to automatically move the object relative to the camera and 
capturing images at multiple known robot positions. If when locating the features 
a sufficient number of features are not found in the field of view of the camera, the 

1 0 relative position and/or orientation of the object is changed until sufficient features 
are found. Prior to communicating the object location to the robot, the necessary 
movement of the object relative to the camera is calculated using the transforma- 
tion between the "Object Space" and "Robot Frame" such that the relative position 
and orientation of the object and the camera is similar to that at the time of 

1 5 training; the relative movement is executed as calculated in previous step; and the 
"Object Space -> Sensor Array Space" transformation is found in the same way as 
in step d) ii). 

Where the object is at rest and stationary and the camera is attached 
20 onto the robot such that its position and orientation can be modified by known 
values, the calibration step is accomplished by placing the "Calibration Model" in 
the camera's field of view and using the robot to automatically move the camera 
relative to the "Calibration Model" and capturing images at multiple known robot 
positions. The step of creating the Object Model is accomplished by using the 
25 robot to automatically move the camera relative to the object and capturing images 
at multiple known robot positions. If when locating the features a sufficient 
number of features are not found in the field of view of the camera, then the 
relative position and/or orientation of the camera is changed until sufficient 
features are found. In this case the step of (step b) iii)) is accomplished by using 
30 the robot to automatically move the camera relative to the object and capturing 
images at multiple known robot positions, and wherein step c) iii) is accomplished 
by creating a new frame called the "Object Frame" that is in constant relationship 
with the "Object Space" and sending the "Object Frame" to the robot and training 
the intended operation path relative to the "Object Frame" and step d) iii) is 



- 19 - 



accomplished by computing the "Object Space" inside the "Robot Frame" using 
the transformation between the "Object Space" and the "Sensor Array" and the 
"Robot - Eye Calibration" and calculating and sending the "Object Frame" to the 
robot and executing the robot path relative to the "Object Frame", and 
5 wherein the following steps are preceded by d) ii) and followed by d) iii): 

a. calculating the necessary movement of the object relative to the camera 
using the transformation between the "Object Space" and "Robot Frame" 
such that the relative position and orientation of the object and the camera 
is similar to that at the time of training; 
1 0 b. executing the relative movement as calculated in previous step; 

c. finding the "Object Space -> Sensor Array Space" transformation in the 
same way as in step d) ii), and 

wherein step d) i) the search area for some of the features is based upon the 
position and orientation of some other features. Prior to communicating the 
object location to the robot, the necessary movement of the object relative 
to the camera is calculated using the transformation between the "Object 
Space" and "Robot Frame" such that the relative position and orientation 
of the object and the camera is similar to that at the time of training; the 
relative movement is executed as calculated in previous step; and the 
"Object Space -> Sensor Array Space" transformation is found in the same 
way as in step d) ii). 
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Where the object is in a robot's grasp such that its position 
25 and orientation can be modified by known values and the camera is 

attached onto another robot such that its position and orientation can be 
modified by known values, the calibration step is accomplished by placing 
the "Calibration Model" in the camera's field of view and using the robot 
to automatically move the camera relative to the "Calibration Model" and 
30 capturing images at multiple known robot positions. The step of creating 

the Object Model is accomplished by changing the relative position of the 
object and camera using movement of one or both robots and capturing 
images at multiple known robots' positions. If when locating the features 
a sufficient number of features are not found in the field of view of the 
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camera, then the relative position and/or orientation of the camera and/or 
object is changed until sufficient features are found. Prior to communicat- 
ing the object location to the robot, the necessary movement of the object 
relative to the camera is calculated using the transformation between the 
"Object Space" and "Robot Frame" such that the relative position and 
orientation of the object and the camera is similar to that at the time of 
training; and the relative movement is executed as calculated in previous 
step; and the "Object Space -> Sensor Array Space" transformation is found 
in the same way as in step d) ii). 

Various means are possible for communicating the object 
location data to the robot. The "Object Space " may be communicated to 
the robot and the intended operation path trained relative to the ''Object 
Space" and then communication of the object location to the robot is 
accomplished by computing the "Object Space" inside the "Robot Frame" 
using the transformation between the "Object Space" and the "Sensor 
Array" and the "Robot - Eye Calibration" and sending the "Object Space" 
to the robot and executing the robot path relative to the "Object Space". 
Alternatively the "Object Space" may be memorized and step d) iii) is 
accomplished by calculating the transformation between the memorized 
"object space" and the current "object space" and communicating this 
transformation to the robot to be used for correcting the operation path 
points. Or the "Object Space" may be memorized and step d) iii) is 
accomplished by calculating the transformation between the memorized 
"object space" and the current "object space" and using this transformation 
to modify the robot operation path points and communicating the modified 
path points to the robot for playback. 

As will be apparent to those skilled in the art in the light of the 
foregoing disclosure, many alterations and modifications are possible in the 
practice of this invention without departing from the spirit or scope thereof. 
Accordingly, the scope of the invention is to be construed in accordance 
with the substance defined by the following claims. 



